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Dedicated to Sigurdur Helgason on the occasion of his 85th birthday 

Abstract. We have in [T] proposed a definition of cusp forms on semisimple 
symmetric spaces G/H, involving the notion of a Radon transform and a 
related Abel transform. For the real non-Ricmannian hyperbolic spaces, we 
showed that there exists an infinite number of cuspidal discrete series, and 
at most finitely many non-cuspidal discrete series, including in particular the 
spherical discrete series. For the projective spaces, the spherical discrete series 
are the only non-cuspidal discrete series. Below, we extend these results to 
the other hyperbolic spaces, and we also study the question of when the Abel 
transform of a Schwartz function is again a Schwartz function. 



1. Introduction 

We initiated, in joint work with Henrik Schlichtkrull, in {Q a generalization of 
Harish-Chandra's notion of cusp forms for real semisimple Lie groups G to semisim- 
ple symmetric spaces G/H. In the group case, all the discrete series are cuspidal, 
and this plays an important role in Harish-Chandra's work on the Plancherel for- 
mula. However, in the established generalizations to G/H, cuspidality plays no role 
and, in fact, was hitherto not defined at all. 

The notion of cuspidality relates to the integral geometry on the symmetric space 
by using integration over a certain unipotent subgroup N* C G, whose definition 
is given in pQ. The map / i— > J N , f(-nH)dn, which maps functions on G/H to 
functions on G/N* , is a kind of a Radon transform for G/H. A discrete series is 
said to be cuspidal if it is annihilated by this transform. 

Let p, q denote positive integers. The Radon transform, and the question of 
cuspidality, on the real hyperbolic spaces SO(p, q + l) e /SO(p,q) e , was treated in 
detail in pQ. We showed that there is at most a finite number of non-cuspidal 
discrete series, including in particular all the spherical discrete series, but also 
some non-spherical discrete series. The non-spherical non-cuspidal discrete series 
are given by odd functions on the real hyperbolic space, which means that they do 
not descend to functions on the real projective hyperbolic space. 

In the present paper, we consider the projective hyperbolic spaces over the clas- 
sical fields F = K,C,H, 

G/H = 0(p + l,q+ l)/(0(p + 1, q) x 0(1)), U(p + 1, g + l)/(U(p + 1, q) x U(l)), 

Sp(p + l, (? + l)/(Sp(p + l, (Z )xSp(l)), 
for p > 0, q > 1. Notice the change of indices from p to p + 1, to simplify formulae 
and calculations. 
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Our main result, Theorem 16. 11 states that the non-cuspidal discrete series for the 
projective hyperbolic spaces precisely consist of the spherical discrete series. The 
Radon transform of the generating functions is also given explicitly. Finally, we 
show that the Abel transform maps (a dense subspace of) the Schwartz functions 
on G I H perpendicular to the non-cuspidal discrete series into Schwartz functions. 
The latter result also holds for the non-projective real case, and is a new result for 
all cases. 

Our calculations and main results are also valid, with p = 0, q = 1 and d = 
8, for the Cayley numbers O, corresponding to the exceptional symmetric space 
F 4 (_2o)/Spin(l, 8). Although the model for this space, and the group action on it, 
is more complicated, this space can for our purposes be viewed as 

F 4 ,(_20)/Spin(l, 8) = "U(l, 2; 0)/U(l, 1; O) x U(l; O)" . 

We state our results in full generality, but only give complete proofs for the non- 
exceptional projective spaces, with some remarks on the other cases in the last 
section. 

We would like to thank Henrik Schlichtkrull for input and fruitful discussions, 
which in the real case lead to the explicit formulae involving the Hypergeometric 
function. We also want to thank Job Kuit for discussions of part (vi) of Theorem l6.1[ 
explaining how to prove a similar result in split rank one, using general theory. 

Part of this work was outlined by the first author at the Special Session 'Radon 
Transforms and Geometric Analysis in Honor of Sigurdur Helgason's 85th Birth- 
day', at the 2012 AMS National Meeting in Boston. He is grateful to the organizers 
Jens Christensen, Fulton Gonzalez, and Todd Quinto, for their invitation to speak, 
and the hospitality at the meeting, and the subsequent Workshop on Geometric 
Analysis on Euclidean and Homogeneous Spaces. 

2. Model and structure 

Let F be one of the classical fields K, C or H, and let x i— > x be the standard (anti- 
) involution of F. We make the standard identifications between C and R 2 , and 
between EI and R . Let p > 0, q > 1 be two integers, and consider the Hcrmitian 
form [■, ■] on F p+9+2 given by 

[x,y] = xii^H \-x p+1 y p+1 ~x p+2 y p+2 x p+1+q+1 y p+1+q+1 , (x, y £ F p+9+2 ). 

Let G = U(p + 1, q+ 1;F) denote the group of (p + q + 2) x (p + q + 2) matrices over 
F preserving [•,•]. Thus U(p + 1, q + 1; R) = 0(p+l,g + l), U(p+l,g + l;C) = 
U(p + 1, q + 1) and U(p + 1, q + 1;H) = Sp(p + l,q+ 1) in standard notation. Put 
U(p;F)=U(p,0;F). 

Let Xq = (0, . . . , 0, 1) T , where superscript T indicates transpose. Let H — V(p + 
l,g;F) x U(1;F) be the subgroup of G stabilizing the line F • x Q in W +q+2 . An 
involution a of G fixing H is given by a(g) = JgJ, where J is the diagonal matrix 
with entries (1, . . . , 1, — 1). The reductive symmetric space G/H (of rank 1) can be 
identified with the projective hyperbolic space X = X(p + 1, q + 1;F): 

X = {z € ¥ p+q+2 : [z, z\ = -1}/ ~, 

where ~ is the equivalence relation z ~ zu, iieF. 

The Lie algebra g of G consists of (p + q + 2) x (p + q + 2) matrices 

= ( B * c)> 
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where A is a skew Hcrmitian (p + 1) x (p + 1) matrix, G is a skew Hcrmitian 
(g + 1) x (q + 1) matrix, and B is an arbitrary (p + 1) x (g + 1) matrix. Here i?* 
denotes the conjugated transpose of B. 

Let if = K\ x if 2 = U(p + 1;F) x \J(q + 1;F) be the maximal compact sub- 
group of G consisting of elements fixed by the classical Cartan involution on G, 
9(g) — .9 € G. Here g* denotes the conjugated transpose of g. The Cartan 

involution on g is given by: 6(X) = —X*. Let g = fi © p be the decomposi- 
tion of g into the ±l-eigenspaces of 9, where t = {X G g : = X} and 
p = {X G g : #(^Q = —X}. Similarly, let q = fj ® q be the decomposition of 
g into the ±l-cigenspaces of a = JXJ, where f) = {X G g : cr(X) = X} and 
q = {X e g : (t(X) - -X}. 

We choose a maximal abelian subalgebra o q C p fl q as 

f / h \ 

a q = I X tl = P . 9 : *! e 
[ V *i o / 

where Pj9 is the (p+<z) x (p+q) null matrix. The exponential of X tl , a tl = exp(X tl ), 
is given by 

(coshii sinhti 
I p , q 
sinhti coshti 

where I p ^ q is the (p + q) x (p + q) identity matrix. Also define A q = exp(a q ). 

Let = {a tl : ii > 0}. Let a(x) = a(kah) = a denote the projection onto 

the A q component in the Cartan decomposition G = KA q H of G. Let M be the 
centralizcr of X\ G a q (i.e., when t\ = 1) in K fl i? . Then M is the stabilizer of the 
line F(1,0, ... ,0, 1), and the homogeneous space K/M can be identified with the 
projective image Y = Y p +i )9 +i of the product of unit spheres S p x 

Y = {y G F^+ 2 : \ Vl \ 2 + ■ ■ ■ + \y p+1 \ 2 = \y p+2 \ 2 + • • • + \y p+q+2 \ 2 = 1}/ ~ . 

The image of the set {z G W +q+2 : [z, z] = — 1, (z\, . . . , 7^ 0} in X is an open 

dense subset, which we will denote by X'. The map 

K/M xM+4X, (fcM.ti) i-> fca tl i2", 

is a diffcomorphism onto X'. 

We introduce spherical coordinates on X as the pull back of the map: 

x(h,y) = (usinh^i; wcosh^i), ti G K+, y = (u;v) e§ p x § 9 . 

We define a (iT-invariant) 'distance' from x G X to the origin as |x| = |x(ii,y)| = 

|ti|. ThenX' = {x G X| |x| > 0}. Wc note that cosh 2 (|x|) = |x p+2 | 2 H V\x p+q+2 \ 2 . 

For g G G, we define \g\ — \gH\. 

Let r = min{p, q}, and let Xt be the (r + 1) x (r + 1) anti-diagonal matrix with 
entries t = (ti, . . . , i r +i) G M r+1 , starting from the upper right corner. We extend 
o q (viz. as t 2 = ■ ■ ■ = t r +i = 0) to a maximal subalgebra a C p as 

X t = ( 
V X t 

We will also consider the sub-algebra dj, = a fl fj = {X t G a : ti = 0}. 
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Let (considered as row vectors) 

u = (tti , . . . , Up) £ ¥ p and v = (v q , . . . , v\ ) £ ¥ q . 

It turns out to be convenient to number the entries of v from right to left as 
indicated. Let furthermore w £ ImF (i.e., w = for F = R). Now define N UjVtW £ g 
as the matrix given by 



u,v,w — 



— w 


u 


V 


w 


\ 


—T 

—u 








—T 

U 




—T 

V 








—T 

— V 




—w 


u 


V 


W 


J 



Then exp(N UtViW ) =1 + N UiV>w + 1/2NI v w , and 

(2.1) exp(Ar w ) • x Q = (l/2(\u\ 2 - \v\ 2 ) + w, u T ; -v T , 1 + l/2(\u\ 2 - \v\ 2 ) + w) T . 
A small calculation also yields that 

(2.2) a tl exp(7V Mj „ iU ,) • x = 

(sinhii + l/2e tl (|u| 2 - \v\ 2 ) + e ll w, u T ; -v T , coshti + l/2e tl (\u\ 2 - \v\ 2 ) + e* 1 w) T , 
for any t\ £ R. 

We note that [X tl ,N u , vfi ] = hN U}Vfi , and [X tl ,N ,o, w ] = 2tiJV , ,, o . Lct 7 (A tl ) = 
ti. Then the root system S 9 for a q is given by T, q = {±7}, for F = R, and 
S g = {±7} U {±27}, for F = C, H. The associated nilpotcnt subalgebra n q is given 
by n q = ^ = {N u>vfi :ue¥P lV e F«}, when F = R, and n q = fp + g 27 = {N u>VtW : 
u £ ¥ p , v £ ¥ q , w £ ImF}, when F = C, H. Half the sum of the positive roots, 
p q = I X)aeE+ m " a i where m a is the multiplicity of the root a, is thus 

( Pq ,X tl ) = ±(dp + dq + 2(d-l))t 1 , 

where d = diniR F. Using the identification A q ~ R, we will also sometimes use the 
definition p q = \ {dp + dq + 2{d - 1)) £ R. 

The (restricted) root system E for a is given by {±ti±tj}, i 7^ j, i,j £ {1, . . .r + 
1}, {±U}, i £ {1, . . . r + 1}, if p ^ q, and {±2tJ, i e {1, . . . r + 1}, if d > 2. Let 
ai,j{X t ) =U+tj,i< j, Pi,j(X t ) =U-tj,i< j, and -fi(X t ) = U. 

We choose two sets of positive roots 

which corresponds to the (standard) ordering ti > t 2 > ■ ■ ■ > t r+ \, and 

£+ = {ttij, 7i, 2 7i } U {ft,^ : i ^ 1} U {-^j : * = 1}, 

which corresponds to the ordering t 2 > £3 > • • • > tr+i > t\. The double roots 
{±27^} are not present for F = R, the single roots {±7i} are not present when 
p = q. The associated nilpotent subalgebras are denoted by n and rii respectively. 
The half sum of positive roots pi with regards to is given by (restricted to A q ) 

(p 1 ,X tl ) = ±((\dp-dq\+2(d-l))t 1 . 

As before, we will sometimes use the definition p x = ^((\dp — dq\ + 2(d — 1)) £ R. 

We note that 7 £ is the restriction of the roots {a^i+j, 71, with 
j £ {1, . . . r}, where 

9 aiJ+1 = {N UtVfi : uj = -Vj, u t = v, = 0, i ^ j}, 
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g/3i,j+i = {N u>v<0 : Uj = Vj, Ui = Vi=0,i^ j}, 

and, for p > q, 

71 = {N u<Vi0 ■ u — (0, . . . ,0,u q+1 , . ..Up), v = 0}, 

which for p < q becomes u = 0, v = (v q , v q -\, . . . , 0, . . . , 0). 

Define n* = ni n n q as the subalgebra associated to the roots {ot\ t \+j, 71, 271}. 
Then, for p > q 

(2.3) n* = {N u , ViW : u = (-W, u'), v e F«, u' £ W p ~ q }, 
and, for p < q, 

(2.4) n* = {N u<VtW : v = »')» « e F", «' G F 9_p }, 

where u r ,v r means that the order of the indices is reversed. In the following we 
shall by abuse of notation leave out the r . 

Remark 2.1. We have the identity £+ = {a € S + : a| 0l] > 0}| ai] . We then have the 
disjoint union £+ = E ++ U E +0 U E -1 , where the second sign refers to ai a . The 
choice of the nilpotent subalgebra n* can thus be described by the correspondence 

3. The discrete series 

From 3 , Section 8] and [4j Table 2], we have the following parametrization of the 
discrete series for the projective hyperbolic spaces, with an exception for q = d = 1: 

{T\ I A = -(dq - dp) - 1 + ia x > 0, /i A G 2Z}. 

The spherical discrete series are given by the parameters A for which fix < 0, includ- 
ing the 'exceptional' discrete series corresponding to the (finitely many) parameters 
A > for which fix < 0. We notice that spherical discrete series exists if, and only 
if, d(q — p) > 2. For q = d = 1, the discrete series is parameterized by A e R\{0} 
such that |A| + p q £ 2Z, and there are no spherical discrete series. 

The parameter A is, via the formula A/ = (A 2 — p 2 ,)/, related to the eigenvalue 
of the Laplace-Beltrami operator A of G/H on functions / in the corresponding 
representation space in L 2 (G/H) (with suitable normalization of A). Using [4j 
Theorem 5.1] (see [U Proposition 3.2] for more details), we can explicitly describe 
the discrete series by generating functions tpx as follows. Let s = si € K describe 
the elements a s = a si G A q . Let A be a discrete series parameter. For fix > 0, we 
have 

i>x(ka s H) = ^ x (x(s,y)) = MA (/c)(cosh s)- x - p " , 

where 4>^ x is a K n 77- invariant zonal spherical function, in particular </>q = 1. For 
/i> = —2m < 0, we have 

tpx(ka s H) = P x (cosh 2 s) (cosh «) -*-'<-*», 

where Pa is a polynomial of degree m. For g = d = 1, we have ifix(ka s H) = 
m (/c)(cosh ,s) _ l A l _ ' 3 i 7 where m = e lm ' is an exponential function with weight 
m = A ± p q , where the sign is determined by the sign of A. 
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4. Schwartz functions 

In this section we recall some results from [2j Chapter 17] regarding L 2 -Schwartz 
functions on G/H. Let S denote Harish-Chandra's bi-i-T- invariant elementary 
spherical function ipo on G, and define the real analytic function : G/H — ► M. + 

by 

Q(x) = y/Sixaix)- 1 ) (x G G). 

We notice that there exists a positive constant C, and a positive integer m, such 
that 

(4.1) a- p " < 6(a) < Car"" (1 + \a\) m , (a G 

Here we use the definition a A = e ( X ' l ° sa \ for a e A+, A € a q £. 

The space C 2 {G/H) of L 2 -Schwartz functions on G/H can be defined as the 
space of all smooth functions on G/H satisfying 

sup e-^Xl + Niri/CAaOl <oo, 

xEG/H 

for all n € N U {0} and D G U(g). 

Let / G C 2 (G/H). Let S C G be a compact set. Then, for any neNU {0}, 
there exists a positive constant C, such that 

(4.2) \f(g-x)\ <Ce(a(x))(l + \x\)~ n , (g£S,xeG/H). 

5. A Radon transform and an Abel transform 

Let N* = exp(n*) and Ni = exp(tii) denote the two nilpotent subgroups gener- 
ated by n* and ni respectively. For functions on G/H we define, assuming conver- 
gence, 

(5.1) Rf{g) = I f(gn*H)dn* (g G G). 

J N* 

Let H A denote the centralizer of A in H. Then Rf(grti) = Rf(g), m <E H A , and 

Theorem 5.1. Let f G C 2 (G/H). 

(i) The integral defining the Radon transform R converges uniformly on com- 
pact sets. 

(ii) Rf G C oc (G/H A N 1 ). 

(iii) The Radon transform is G- and Q-equivariant. 

Proof. We first assume p > q. Let / G C 2 (G/H), and fix a compact set S C G. Let 
neff. Then 

(5.2) / \f(gn*H)\dn* <C [ o(n*)-'< (1 + \n*\y n+m dn* (g G S), 

J N* JN* 

for the constants C and m given by (|4. 1 [) and (|4.2p . From (12.11) and (|2.3[) . we have 

cosh 2 (| exp(W (WiU , )it)ill ,)|) - (1 + 1/2| U '| 2 ) 2 + |«| 2 + M 2 . 

Using that logs < arccoshs < log s + log 2, when s > 1, we see that the last integral 
in (|5.2I) is bounded by 

c/ ((i + i/2K| 2 ) 2 + M 2 + H 2 r i£± ^ li 

x (1 + log((l + l/2|w'| 2 ) 2 + \v\ 2 + \w\ 2 ))- n+m du'dvdw, 
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where C is a positive constant. 

Consider the integral (x E R fe , y € R z ), with n > 2, 

/ (1 + M 4 + \y\ 2 )- a (l + log(l + \x\ 4 + \y\ 2 ))- n dxdy. 

JR k xW 

With the substitution y = y/l + \x\ 4 z e I 1 , we get 
/ (1 + \x\ 4 )~ a+L i (1 + |z| 2 )- Q (i + log(l + |z| 4 ) + log(l + \z\ 2 ))- n dxdz < 
/ (l + |.x| 4 )- a +5(l + log(l + |x| 4 ))-t ( i x f (l + |z| 2 )- a (l + log(l + |^| 2 ))-^dz, 

which is finite if, and only if, k < 4a — 21 and I < 2a. 

We have k = dp — dq, I = dq + d — 1 and a — (dp + dq + 2(d — l))/4, whence 
k = 4a — 21 and I < 2a, and the integral (|5.1J converges uniformly on compact sets. 

In the p < q case, we see from (I2.1[) and (|2.4I) . that cosh 2 (| exp(A f u ( tlil ,/) t0 )|) = 
|w'| 2 + \u 2 \ + (1 - l/2|w'| 2 ) 2 + |w| 2 = 1 + M 2 + 1/4H 4 + |u;| 2 , and we proceed as 
before, reversing the roles of u and v. □ 

We define the Abel transform A by Af(a) — a Pl Rf(a), for a E Aq. 

Theorem 5.2. Let g E G and f E C 2 (G/H). Let A denote the Laplace-Beltrami 
operator on G/H and let denote the Euclidean Laplacian on Aq. Then 

(5.3) A(Af) = (A Aq - p 2 q )Af (oeA,). 

Proof. See [U Lemma 2.4], and the discussion before and after this lemma. □ 

Let ip\ belong to the discrete series with parameter A. Since Aip\ = (A 2 — pVfipx, 
we see that Aip\ is an eigenfunction for the Euclidean Laplacian A^ on Aq with 
eigenvalue A 2 . This implies in particular that s H > Rip\(a s ) is a linear combination 

of e (X-pi)s and e (-A- PI )s_ 

6. The main result 

Here we state the main theorem, to be proven in the following sections. We will 
in particularly be interested in the values of Rf on the elements a s E A q , so for 
simplicity we write Rf(s) = Rf(a s ), and, similarly, Af(s) — Af(a s ). 

Let R > 0, and let Cr(G/H) denote the subspace of smooth functions on G/H 
with support inside the (K- invariant) 'ball' of radius R. Let similarly C£?(R) denote 
the subspace of smooth functions on R with support inside [—R,R]. Finally, let 
6>(R) denote the Schwartz functions on R. 

Theorem 6.1. Let G/H be a projective hyperbolic space over R, C, H, with p > 
0, q > 1, or over O, with p = 0, q = 1. 

(i) If d(q — p) < 2, then all discrete series are cuspidal. 

(ii) // d(g — p) > 2, then non-cuspidal discrete series exists, given by the pa- 
rameters A > with p\ < 0. More precisely, if =^ / E C' 2 (G/H) belongs 
to T\, then Af(s) = Ce Xs , with C ^ 0. 

(hi) T\ is non-cuspidal if and only if T\ is spherical. 

(iv) Ifp > q, and f E C^{G/H), for R > 0, then Af E C%> (R). 

(v) Ifd(q-p)<l, and f EC 2 {G/H), then Af E 5(R). 
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(vi) Assume d(q — p) > 1. Let D be the G -invariant differential operator 
Ap(A p — A 2 ) . . . (A p — X 2 ,), where Ai, . . . , A r are the parameters of the non- 
cuspidal discrete series, and A p = A + p 2 . Then A(Df) G <S(R), for f in 
a dense subspace ofC 2 (G/H). 

Remark 6.2. The theorem also holds for the non-pro jective spaces SO(p + l,q + 
l) e / SO{p + l,q) e , except for item (iii), due to the existence of non-cuspidal non- 
spherical discrete series, corresponding to the parameters A > 0, with G 2Z + 1 
and [i\ < 0. 

Remark 6.3. The conditions in item (vi) essentially state that Af is a Schwartz 
function if / is perpendicular to all non-cuspidal discrete series. The factor A p , 
however, cannot be avoided, except in the cases d = 1 and q — p odd. 

Remark 6.4. For the exceptional case, only (ii), (iii) and (vi) are relevant. The 
spherical discrete series corresponds to A = 3 (p\ = 0) and A = 1 = —2). 



7. Proof of the main theorem for p > q 
Proposition 7.1. Let p > q. 

(i) Let / <= C%(G/H), for R > 0. Then Af G C£?(R). 

(ii) Let / G C 2 (G/H). Then Af G 5(R). 

Proof. Let / G C%(G/H), for R > 0. By (gU and (|2"3]) . we have 

cosh 2 (|a s exp(^ !tl i )i , )!tu )|) = (coshs + l/2e s \u'\ 2 ) 2 + \v\ 2 + \e s w\ 2 > cosh 2 s, 

and thus Rf(s) = 0, for |s| > R, which shows (i). 

For (ii), let / G C 2 {G/H). As before we have, for n G N, 

(7.1) f \f(a s n*H)\dn* <C I a{a s n*)- p " (1 + \a s n*\)~ n dn* , 

Jn* Jn* 

where C is a positive constant. 

The integral in (|7.ip is bounded by 

/ ((coshs + l/2e>'l ) 2 + \v\ 2 + \e s w\ 2 ) ^ 1 

x (1 + log((coshs + l/2e s \u'\ 2 ) 2 + H 2 + \e s w\ 2 ) 1/2 ))- n du'dvdw 

< (coshs)- ""^ 2 '"' 1 ' f (l + (l/v / 2(cOshs)-5 e ^/2| u '|)2)2 

i q 19 cip + dq + 2(d — 1) 

+ |(coshs) _i -yr + |(coshs)~Vi«| 2 ) J 
x (1 + log((l + (l/V2(coshs)-5 e «/2| M '|)2)2 

+ |(coshs) _1 i;| 2 + | (cosh sy 1 e s w\ 2 y n du'dvdw, 

since log coshs > 0. 

Consider the substitutions u = l/v / 2(coshs) _ 2e s / 2 u', v — (coshs)^ 1 ?; and w = 
(coshs)~Vw. Then du' = (\/2(cosh s)h- s / 2 ) dp - dq du, dv = (cosh s) d «dw, and 
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dw = ((coshs)e s ) d 1 dw, and the above integral becomes 

i—dp-dq dp-dq + 2(d-l) f „ „ „ ,n, dp + dg + 2(d- 1) 

= V2 e 5 s ((1 + \u\ 2 ) 2 + \v\ 2 + \w\ 2 ) J 

x (1 + log((l + |u| 2 ) 2 + \v\ 2 + \w\ 2 ))- n dudvdw 

where C p ^ q is a constant only depending on p and q. The theorem follows using the 
[/(g)-equivariance of the Radon transform from Theorem l5.1l (iii). □ 

Let C 2 (G/H) d = L 2 {G/H) d n C 2 (G/H) denote the span of the discrete series in 
C 2 {G/H). 

Proposition 7.2. Let p > q. Then Rf = 0, for / G C 2 {G/H) d . 

Proof. Let / G C 2 {G/H) d . Then .A/ belongs to S(A q ) by Theorem O but at the 
same time Af is also an eigenfunction of Aa„ on A q . We conclude that Af = 0, 
and thus Rf = 0. ' □ 



8. Proof of (i) - (v) of the main theorem for q > p 

Let be a generating function for the discrete series with parameter A. We 
notice that — A — h(dq — dp) + 1 = A + p q — dq — (d — 1) + 1. 

Proposition 8.1. Let p < q. Then i?V>A = 0, for fi\ > 0. For < 0, we have 
(8.1) Rtp x (a s )= I ip x {a s n*)dn* = Ce^~^' , (s G M), 



where C ^ is a constant depending on p, q. 

Proof. We define a if-invariant function ip\ as ip\(ka s H) = (coshs)~ A-P(| . Then 
by (|2.2p and (|2.4p . the Radon transform Rtp\(a s ) is 



J<J-d PxK tip xR £i-l 



((coshs - l/2e>'| 2 ) 2 + \v'\ 2 + \u\ 2 + \e s w\ 2 )~^dv'dudw. 



Substituting v' — (e s / (2 coshs) 1 / 2 ?/, u = (l/coshs)u and w = (e s / cosh s)u>, this 
becomes 



1 \ -dp / £ s N 



2 cosh s J V cosh s / V cosh 



l s 



(cosh s) 



-A-p,, 



(1-|^| 2 ) 2 + f - 2 ) \v'\ 2 + \u\ 2 + \ti\ 2 ) 2 dS'dfiui 

dq-dp X R d P XE d_1 V \ e cosn S 

_ dq—dp dg-dp + 2(d-l) . 

2^~ e ^ s (coshs)~ A 

; / (l + |S'| 4 - 2(tanhs)|C'| 2 + \u\ 2 + H 2 )~^ di'duw. 

J R dq-dp x R dp X Rd - 1 
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Using the substitution u = (1 + Iv 1 ] 4 — 2(tanhs)|v'| 2 ) 1 ' 2 u, and likewise for w, 
the integral becomes 

X + Pq | dp | d — 1 

L~.l|4 „M„~.'|2\ 2 ^ — + ~2- 



/ 



(1 + l^l 4 - 2(tanhs)|t7 

: <!- <i !'xR d PxR d - 1 



x (1 + |i2| 2 + \ti\ 2 ) dv' (Md'Si 

pOO A + Pt| -dp-(d-l) 

C (l+£ 4 -2(tanhs)£ 2 ) 2 ^q-dp-i^ 

Jo 

1 „ f°° i , s s _ ±hga^^rlj dq - dp 1 



= -C / (1 + x 2 - 2(tanhs)x) 2 l cfcc 



using polar coordinates, where C is the positive constant given by 

C= / (l + ry 2 + cr 2 ) ^ -q^' 1 drja^ 2 da < oo. 

Jo Jo 

From [5, 3.252(10)], we get 

(l + x 2 + 2(cos t)x) ~" x^ x dx = 

2 v -*(smt)?- v T{v + -)B(ji, 2v - [i)P*~ v , (cost). 

We also have 

1, + ^3-") / ! 1111 

Vr^ + ^^J 2 F 1 ^ + , + -, M -, + -;,+ -;--- y 

With y = cos t = — tanh s, for < t < tt, we get sin t = 1/ cosh s, 1 — y — e s / cosh s 
and 1 + y = er s j coshs. Putting this together, we get 

(1 + x 2 - 2(tanhs):r)~'V~ 1 d:E = B(fi,2is - (j,)(2(cosh.s)e s ) u ~^ 

/ 1 111 



2'" ' 2'" ' 2' 1 + e- 2s 
With /i = and „ = A+ Pq -rfp-(d-i) ^ we get 

RMas) = C A e--(l + e--)-^ 2 ^ 1 - ■ ^ + * " *■ 1 



2 ' 2 ' 2 ' 1 + e- 



2* 



where Ca is a positive constant depending on p, q and A. 

The hypergeometric function z i— )- 2-Fi(a*a/2, 1 — A*a/2; (/ia + rfg — dp)/2; z) is a 
polynomial of degree —[i\/2 for ^a < 0, and degree fj,\/2 — 1 for /ia > 0. We thus 
immediately get (|8.1[) for ^a = 0. 

Now let /ia = — 2rn < 0. We can write i/j\(a s ) as the sum (coshs)~ A ~ P(| + 
E™ i Q(cosh S )-( A + 2 ^-^; or Va = ^a + Ejli Q^a+ 2j - It follows that i?^ A (a s ) 
can be written as a sum C e- ds (l + e~ 2s ) m + Y!?=o Q e ~ ds (! + e~ 2s ) j , where C 
is a non-zero constant corresponding to the factor ij)\{a s ) — (cosh s)~ A ~ p <> . Thus, 
since we know that Rip\(a s ) is a linear combination of e' A ~ Pl ' s = e( pA_d ) s and 
e (-A-pi) Sj we get Ripx{a s ) = Ce^ x ^ d)s , for a non-zero constant C. 
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Let finally fix > 0. Then |?/>a| < \\(f>^ x \\oo^x, and I-R^aI < ||<?W \\ooRi>\- We have 

(8.2) \R*l>\(a a )\ < CiR^xias) < C 2 e~ ds for s -> oo, 
and 

(8.3) \R*px(as)\ < Crf^M < C 2 e^- d > for s -oo, 

for positive constants Cj. Since s H ► Ripx{a<s) again is a linear combination of 
e (nx-d)s and e (_A- Pl ) S; we gee from (jg^J) a nd ||53J) that fl^A = 0. □ 

Consider the cases where p < q and d(g — p) < 2, i.e., the cases (d,p,q) = 
(1, q — 1, q), (7) = (1, <? — 2, g) and (7) = (2,q — 1, (7). In the first case 
/iA = A + 1/2, and fix = A in the last two cases. This means that fix > and (i) 
follows from Proposition 18. II 

For the proof of (v), we need to consider the cases where p < q and d(q —p) < 1, 
i.e., the cases (d,p,q) = (l,q— l,q)- From [1] Theorem 5.1 (iii)(a)] (recall that in 
that paper p := p — 1), we see that the Schwartz condition in the real case is also 
satisfied for p = q — 1 . 

9. Reduction to the real case (d = 1) 

Some of our results above for the projective hyperbolic spaces could also be 
established from p] Theorem 5.2] via the remark below. However, we feel that the 
new and different presentation, and in particular the new proof of Proposition 18. 1[ 
merits the space given. 

Let F = C, H, with p > 0, q > 1, and d = dirriR F. There is a natural projection 

X(dp,dq,R) -> X{p,q,¥), 

with a natural action of U(l; F) on X(dp, dq; K). Let 

£a(p,9,F) = {/ G C°°(X(p,q,F)) I A/ - (A 2 - Pq)/}, 

then there is a G-homomorphism, 

£a(p,<Z,F) -^(dp.dg.R), 

which is an isomorphism onto the U(1;F) invariant functions in £x(dp,dq,M). We 
refer to [7] for more details. 

Letp'+l = d(p+l) andg'+l = d(q+l). We note that p q = \(dp+dq+2(d-l)) = 
j(p' + q') = p' q . Let xpx be as in the proof of Theorem 18. 11 and let R d q and R^, , 
denote the Radon transforms corresponding to the spaces X(p + l,q + 1,F) and 
X(p' + l,q' + 1,M) respectively. Using the substitution w' — e s w, we get the 
identity: 

(9.1) RtJx{a s ) = e-^-^R^^Mas), (s G E), 

which shows that some of our results for the projective hyperbolic spaces follow 
from the real (d = 1) case. Similarly, we get 

A d p ^x =Al, <q ,i>x- 
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10. Proof of (vi) of the main theorem - A Closer study of Rf near 

+00 

We want to prove that A(Df) € S(K), for / e C 2 {G/H) and d(q - p) > 1. 
Although we believe this to be true in general, our proof, near +00, is only valid 
for the dense G- invariant subspace generated by the If -finite and (if HH) -invariant 
functions. 

For d = 1, pfl Theorem 5.1 (iii)(b)] yields that the Schwartz decay conditions are 
satisfied near —00 for A(f), and thus also for A(Df). For d > 1, the proof from 
[I] is easily adapted in the same way as formula (|9.1[) . which leaves us to study Rf 
near +00. We will concentrate on the proof for d = 2, 4 below, with some comments 
on the d = 1 case, and further remarks in Section ITTl 

Consider the subgroup T given by 

( I P +i \ 
cos6> sin 6» 

Jg_l 

\ -sin^ cos6> ) 

where Ij denotes the identity matrix of size j. Then 

kga t ■ xq — (sinhi, 0, . . . , 0; sin 9 cosh t, 0, . . . , 0, cos 9 cosh£). 

We see that H D if x , with K x normalizing T, and if 2 = (K 2 n H)T{K 2 n H), 
where if 2 D H = U(q,¥) x U(l,¥). Furthermore U(q,¥) centralizes A, and as is 
easily seen, (if n H)kgwatH = (K n H)kga t H, for w € {7(1, F). From this we 
deduce that 

(10.1) KnH = (K C\H) T {K nH) A « and G — (K C\ H)TAH, 

where (K n H) T and (K n if)' 4 '' denote the centralizers of T and A q in K D H 
respectively. It follows that a K D iJ-invariant function is uniquely determined by 
the values f(k e a t H), for (6>,i) G [0,tt] x R+. 

From the equation (if n H)kga t H — (K n H)a s n UtV / tW H, we get 

(coshi) 2 = (coshs - l/2e s |i;'| 2 ) 2 + |i/| 2 + |m| 2 + \e s w\ 2 , and 

' (cos6»cosht) 2 = (coshs - l/2e>'| 2 ) 2 + \e s w\ 2 . 

Let x = = \v'\ and z = e s |iy|. Let v = — sinhs + l/2e s y 2 , then ?/ 2 = 

1 + 2e~ s i; — e _2s , and 

(cosh<) 2 = 1 + x 2 + v 2 + z 2 , and 
(10.3) , , , 

(cos 9 cosh t) 2 = (v - er s ) 2 + z 2 . 

For p — 0, the variable x — and the integration over a; disappears, and for d = 1, 
the integration over z disappears, Furthermore, the equations (|10.2p and (|10.3p arc 
slightly different in these cases, see Section ITTl 

Consider a K D i?- invariant function / of irreducible if -type. Then the function 
k h- > f(kcit) is a zonal spherical function on K/{K (1 -ff), a Jacobi polynomial in 
cos 9, of even order. We can thus decompose / as a finite sum of functions of the 
form h(kga t ) = (cos 9) m h(a t ) , where m is even and h is an even function. 

Define an auxiliary function H by ii(cosh 2 i) = (coshi)~ m /i(a t ). Then, using 
the change of coordinates from before, we have: 

h(kga t ) = (cos 9 cosh t) m H(cosh 2 t) = ((« - e~ s ) 2 + z 2 )^H(l + x 2 + v 2 + z 2 ), 
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where for each N € N, 

\H(1 + x 2 +v 2 + z 2 )\ < C(l + x 2 + v 2 + z 2 )-^(\ + log(l + x 2 +v 2 + z 2 ))~ N . 
With the above substitutions, we find 

/•OO />oo />oo 

Rh(s) = e~ ds / / H{l + x 2 +v 2 + z 2 )((v - e- s ) 2 + z 2 )^ 

JO JO J- sinh a 

x (l + 2e' s v-e- 2s ) i ^x a z d - 2 dvdxdz, 

where a = dp — 1 , /3 = d(q — p) — 1 > 0, i.e., /3 is a positive integer. 
We have the following upper bound, for s > 0, since j3 > 1: 



oo poo />OC 



(i + x 2 + v 2 + z 2 y 



\Rh{s)\ < Ce- ds I I I i '-^(l + v 2 +z 2 Y 

' 1 1 , (1 + log(l + X 2 + v 2 + z 2 )) N v 



x (1 + v 2 ^ ^ x a z d 2 dvdxdz<+oo. 



Applying Lebesgue's theorem, we get 

«oo poo />oo 



/•OO /"OO /"OO 

\ime ds Rh(s)= / / H(l + x 2 + v 2 + z 2 ){v 2 + z 2 )^x a z d - 2 dv dx dz. 

s ^°° Jo Jo J-oo 

For convenience, we replace z by u. We can define Rh(s) as a function of the 
variable z = e~ s near 2 = 0, for z > 0. Let F(z) = e ds Rh(s), then 



F(z) = f 
Jo 



OO pOG /"OO 

H(l + x 2 + v 2 + u 2 )((v - zf + u 2 )^ 
x (1 + 2zv - z 2 ) 3 ^ x a u d ~ 2 dv dx du. 



Let ko be the largest integer such that kg < (f3 — l)/2 + 1, and < k < ko. The 
derivatives d k /dz k of the integrand are zero at v = — sinhs = |(z — whence 
the integrand is at least fco times differentiable near z = 0, and we can compute the 
derivatives d k /dz k F(z). For ko > 0, we will use Taylors formula to express F(z) as 
a polynomial of degree fco — 1, plus a remainder term involving d k ° / dz k ° F '(£) , for 
some < £(z) < z. 

Lemma 10.1. Fix v,u € M., m G 2Z + and S £ , and define 

S(z) = S v ^ m , 5 (z) = {(v ~ z) 2 + u 2 )^{l + 2zv - z 2 ) 5 . 

For < j < 6 + 1, d? I 'dz 3 'S(z) is a polynomial in ((v — z) 2 + u 2 ), (v — z) and 
(1 + 2zv— z 2 ), of degree at most m + 5 in v, m in u and m + 25 — j in z. For z = 0, 
the degree is at most m + j in v, and m in u. When j is odd, d 3 /dz J S is an odd 
function of v at z — 0. 

Proof. Straightforward, using that d/dz(l + 2zv — z 2 ) — —d/dz((v — z) 2 + u 2 ) — 
2{v-z). □ 

Note, that for d(q — p) odd, that is, d = 1 and q — p odd, the term (/3 — l)/2 = 
{{l ~ p) ~ 2)/2 is a half- integer, and the statements in Lemma 1 10. II have to be 
changed accordingly. 

Using Taylors formula, we get 

F{z) = c + ax + c 2 z 2 + ... + c ko ^ 1 z k °- 1 + R ko (Oz ko , 
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where < £ < z, and 

-i />00 />00 />OG 7J 

A. I I I . Q Q Q,Ci 



c, = — 



i?(l + a; 2 + ?/ )j7 5 , t ,,«,m,(/3-i)/ 2 (0)a; Q u d 2 dvdxdu, 



f- Jo Jo J-oo dzi 

for j e {0, . . ., ko — 1}. By Lemma liO.ll Cj = 0, for j odd. The remainder term 
R k (0 is given by: 

I r°° r°° r°° ^ko 
— H{\ + x 2 +v 2 + u 2 )— T -S v .^ m . {fi . 1)/2 {£,)x a u d ' 2 dvdxdu. 

kol Jo Jo ./iCf-e- 1 ) dz ' 

Consider Ah(s) = e plS Rh(s) = z~ ( - f>1 ~ d] F(z), which is equal to 

c z- (pi - d) + c 2 z-^- d -V + ... + Cfeo _ 2 z- 2 + Cfe^iZ- 1 + R ko (0- 

The exponents p\ — d — 2j = d(q — p)/2 — l — 2j, for j e {0, . . . , fc — 1}, correspond 
to the parameters Ai, . . . , A r of the non-cuspidal discrete series. From (|5.3p . and 
the definition of the differential operator D in Theorem 16.11 (vi), A(Dh) thus only 
has a possible contribution from the remainder term, and, due to the term d 2 /ds 2 , 
no constant term at oo. 

Note, that for d(q — p) odd, the last two terms are: Ck -iz~i + z^Rk (^), 
where the last term is rapidly decreasing. For the other cases, the constant term 
Cfi fco = lim^oo Rk (e~ s ) could be non-zero, but we will prove that Rk (£) — Cfl fco 
is rapidly decreasing at +oo, where £ = £(s), with < £ < e~ s . We also consider 
the case ko — 0, with £ = e~ s . 

Let G{v,u,z) = l/k \d ko /dz ko S VtU , mti p_ 1)/2 (z)- Then G(v, u, z) - G(v, u, 0) = 
zP(v,u, z), where P is a polynomial of degree less than m + (/? — l)/2 in v, and 
less than m in u. Let \P\ and |G| denote the polynomials defined from P and G by 
taking absolute values in all coefficients. 

Let in the following C denote (possibly different) positive constants. With < 
£ < e~ s , we get the following estimates 

\Rk (Z)-c Rko \ < 

pQO pOO pOO 

e~ s / / \H(l + x 2 +v 2 + u 2 )\\P\{v,u,l)x a u d - 2 dvdxdu 

J0 Jo J-oo 

oo />oo /•— sinh s 

\H(1 + x 2 + v 2 + u 2 )\\G\{v,u,0) x a u d ~ 2 dvdxdu. 



JO 

The first integral is bounded by Ce~ s , since the double integral is convergent. 
The second integral is bounded near infinity by Cs~ N , for all N, which is seen as 
follows. For s large, the integrand is for every N € N bounded by C(x 2 + v 2 + 
u 2)-( P +m)/2| u |m+fc \ g( x 2 +v 2 + u 2 )~ N . Substituting v = -v,x = x'v and u = u'v, 
we have the estimates 

/•OO />00 />OG , 

Jo Jo ./sinh s 

x (log(t; 2 ) + log(l + x' 2 + u' 2 ))- N x' a u' d - 2 dv dx' du' 

p CO 

<C v- p " +ko+a+d (log(v))- N dv. 

J sinh s 
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Inserting the values p q = d(p + q)/2 + d— 1, ko = (d(q—p) — 2)/2, and a = dp — 1, 
we end up with 

/>oo 

C / v- 1 {\og{v)y N dv = C{N - l)- 1 {\og{smhs))- N+1 < C S - N+1 . 

J sinh s 

It follows that Rk (£,)~ Cn ko is rapidly decreasing at +oo, whence A(Dh) is rapidly 
decreasing at +oo, since the constant term is not present, which finishes the proof 
of Theorem 16. II (vi) for K- irreducible (K n i?)-invariant functions. 

Finally, consider the G-invariant subspace V of C 2 (G/H) generated by the K- 
irreducible (K n i/)-invariant functions. The conclusion in (vi) is clearly satisfied 
for / G V. We need to show that V is dense in C 2 (G/H). Let ^ / G L 2 (G/H) 
be perpendicular to V. Let U be the closed G-invariant subspace of L 2 (G/H) 
generated by /. Then U contains a non-zero G°°-vector fi G C 2 (G/H), and after 
a translation, we may assume that fi(eH) ^ 0. The function ji defined by 7^ 
fiigH") = JxnH fii^QH) dk is then a (K n i?)-invariant element in U, belonging to 
the closure of V, which is a contradiction. 

11. Final Remarks - the remaining cases 

Theorem 16.11 also holds for the real non-pro jective space G/H — SO(p + 1, q + 
l) e /SO(p + l,q) e , except for item (iii). The statements (i), (ii), (iv) and (v) are 
proved in [I]. For the proof of (vi), the last equations in (|10.2j) and (|10.3|) should 
be replaced by 

cos#coshi = coshs — l/2e s \v'\ 2 , and 
cos 9 cosh t = (v — e~ s ). 

Then (0,t) £ [0, 2tt] x K+, and m could be odd. For p = 0, the first equations in 
(fT0T2]) and (fT0T3| should be replaced by: 

sinhi = sinhs — l/2e s v' 2 , and 
sinhi = —v, 

with (6,t) e [0,2tt] x E, H defined by H(-smht) = (cosh t)- m h(a t ), and 
|JT(«)| < C(l + v 2 )-^(l + log{l + v 2 ))- N . 

With these remarks it is not difficult to modify Lemma riO.li and complete the proof. 
Notice, that a priori all constants Cj in the Taylor expansion could be non-zero. 

Finally, we consider the exceptional case, with F = O (and p = 0,q = l,d = 8). 
We will show that the formulas (|10.1[) and (| 10 . 2[) are meaningful and true for this 
case as well. The formula ([lO.ljl was already shown to be true and used in 0]. We 
give a brief outline of the proof of (|10.2[) . 

According to [6] and [8] , the exceptional group G can be defined by the automor- 
phisms of a 27 dimensional Jordan Algebra J 12 parameterized by £, u G R 3 x O 3 , 
with basis E x , E 2 , E 3 , Fx, F 2 , F 3 . We denote an element in J lj2 by X(£, u). 

The subgroups H and K are in fact equal to the stabilizers of E 3 , respec- 
tively of Ei. The subgroup N, which in this case equals N* , is defined in [5] as 
u(y, z), y Glm(O) and z G O. The subgroups A = {a t } and T — {kg} are also de- 
fined there. In [B] the expression a s u{y, z)E% is calculated, and in [5] the expression 
kgX(£,u) is calculated; combining these two calculations kgatE^ = kga t u{Q,0)E 3 
can be calculated. 
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Recall that £1,^3 are invariant under K n H. To derive the first formula in 
f|10.2|) . we only need to compare the first coordinates of keatE^ and keatu(y, z)Es, 
£i(s, y, z) = £1 (t, 6); to derive the second formula in (jf 0.2[) . we only need to compare 
the third coordinates of kgatE^ and k$atu{y, z)E$, £$(s,y,z) = £,3(t,9). We have 

a(M) = -(cosh(2i)-l)/2, 

y,z) = - cosh (2a)((l - z)/2 + |z| 4 /4 + \y\ 2 ) 

+ sinh (2s)(l/2|z| 2 (l - |z| 2 /2) - \y\ 2 ) + (1 - |z| 2 )/2, 

&(t, 0) = (cosh (2t) + 1)/4(1 + cos (26>)), 

y, z) = cosh (2s)((l - z)/2 + |z| 4 /4 + |y| 2 ) 

- sinh (2,s)(l/2|z| 2 (l - |z| 2 /2) - |y| 2 ) + (1 - |z| 2 )/2. 

A tedious, but straightforward calculation, leads to the formulas (I10.2[) . with v' 
replaced by z, and w replaced by y. 
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